A speech understanding module for a multimodal mathematical formula editor

As part of a framework for a multimodal mathematical formula editor which will support natural speech and handwriting interaction, a single stage speech understanding module is presented. It is based on a multilevel statistical, expectation driven approach. Completely spoken realistic formulas containing basic arithmetic operations, roots, indexed sums, integrals, trigonometric functions, logarithms, convolutions, fourier transforms, exponentiations, and indexing (among others) were examined. The speaker specific or formula specific structural recognition accuracies reach up to 90 % or 100 %, respectively. For visualization and postprocessing purposes, a transformation into Adobe FrameMaker documents is performed. An advanced variant of this architecture will further be utilized as the basis for a multimodal semantic decoder incorporating combined script and speech analysis. It will enclose a so-called Multimodal Probabilistic Grammar which will be trained via multimodal usability tests.